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Abstract 

The 1 999- Advanced Placement® (AP®) Psychology Exam contains items drawn from 13 factors 
related to the study of psychology. Heretofore, this factor structure had not been explored 
empirically. Therefore, the current study focuses on evaluating the fit of confirmatory factor 
analysis (CFA) models to exam items. Since exam items were dichotomous and polytomous, the 
CFA models were fit to polychoric correlation matrices using weighted least squares with the 
inverted matrix of asymptotic variance/ covariances estimates serving as the weight matrix (W" 1 ). 
A rationale for using this method is provided. The correlations among items, as well as their 
asymptotic variances and covariances, were estimated with PRELIS 2.3 and the CFA was 
performed with LISREL 8.3 (Joreskog & Sorbom, 1999). Results indicated that the proposed 
CFA models fit the data well, which suggests that the theoretical factor structure of the exam is 
plausible. Limitations and next steps are discussed. 
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Measuring Knowledge of Introductory Psychology: What are the Relevant Constructs? 

The College Board’s Advanced Placement Program® (AP®) offers a course and 
examination to qualified students who wish to complete studies in secondary school that are 
equivalent to an introductory college course in psychology (The College Board, 1999a). The 
principals of the AP Psychology course, as indicated by The College Board (1999a), are repeated 
here: 

The purpose of the AP course in psychology is to introduce students to the systematic and 
scientific study of the behavior and mental processes of human beings and other animals. 
Students are exposed to the psychological facts, principals, and phenomena associated 
with each of the sub-fields within psychology. They also learn about the methods that 
psychologists use in their science and practice (p. 3). 

These principals create an excellent learning environment for AP Psychology students. A 
cumulative exam at the end of the school year measures proficiency in knowledge acquired from 
course work. 

The 1999 AP Psychology examination is representative of the content covered in a 
college level introductory course in psychology; it is therefore considered appropriate for 
measuring skills and knowledge in introductory psychology. (The College Board, 1999a). 

Grades on the AP Psychology exam are intended to allow participating colleges and universities 
to award college credit, advanced placement, or both to qualified students (The College Board, 
1999b). 

The AP Psychology exam is theorized to measure 13 content domains, or factors, related 
to the study of psychology. The following is a list of labels for the content domains covered by 
the exam: a) Method, Approaches, History; b) Biological Bases of Behavior, c) Sensation and 
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Perception; d) States of Consciousness; e) Learning; f) Cognition; g) Motivation and Emotion; h) 
Developmental Psychology; i) Personality; j) Testing and Individual Differences; k) Abnormal 
Psychology; 1) Treatment of Psychological Disorders; m) Social Psychology. 

The AP Psychology Examination consists of a 70-minute multiple-choice and a 50- 
minute free response section. The multiple-choice questions comprise two-thirds of a student’s 
grade. There are two free response questions that evaluate “a student’s mastery of scientific 
research principles and ability to make connections among constructs from different 
psychological domains” (The College Board, 1999a). 

The development of the AP examinations involves an extensive process taking several 
years. Committees comprised of secondary school teachers and college professors in cooperation 
with test development experts at ETS have authority over developing the AP tests. Committee 
members bring to their knowledge of the curricula and instructional methods in their fields. As 
faculty members, they know the abilities and skills that are critical to mastery in a given subject, 
and how students can demonstrate them. Faculty consultants score free-response questions. 
Detailed scoring guidelines and various “checks and balances” ensures inter-rater reliability of 
scoring (The College Board, 1 999a) 

Despite careful development of test items and a clear description of their associated 
content domains, it has not been determined empirically whether The College Board’s 1999 AP 
Psychology test measures what it intends to measure. The purpose of the current study is to 
explore the construct validity of the exam through an application of confirmatory factor analysis 
(CFA). Our rationale for undertaking this kind of investigation is based almost directly on 
Messick (1989): 
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Construct validity is evaluated by investigating what qualities a test measures, that is, by 
determining the degree to which certain explanatory concepts or constructs account for 
performance on the test (p. 1 6) 

Factor analysis can be considered one method of investigating the construct validity of a test. 
However, since both dichotomous and polytomous variables comprise the test data, traditional 
factor analytic methods are insufficient for addressing this purpose. 

Problems with factor analyzing dichotomous data were originally identified within the 
field of exploratory factor analysis. The major problem is that spurious item difficulty factors 
tend to be extracted when traditional methods are applied to dichotomous data. Such spurious 
factors were first noticed in the realm of intelligence tests in instances where factor analytic 
solutions yielded several factors with items of similar difficulty grouped together (Gorsuch, 
1983). The restriction in the range of data for dichotomous test items reduces the phi 
correlations among these variables, which results in spurious factors being extracted during 
factor analysis. Similar problems arise when CFA is performed on dichotomous data except that 
instead of spurious factors being extracted, plausible models are rejected in favor of other models 
that are less parsimonious. 

Several researchers have proposed alternative methods for factor analyzing dichotomous 
data. One approach is to create parcels of dichotomous items and then factor analyze the 
correlations or covariances among these parcels (Dorans & Lawrence, 1999). By collapsing 
items into parcels and making them roughly continuous, this procedure corrects for the 
restriction in range in the dichotomous items. To implement this procedure correctly, parcels 
should contain a sufficient number of easy, middle-difficulty, and hard items (Dorans & 
Lawrence, 1999). Researchers should be aware that the parcel approach does not address 




6 



Introductory Psychological Constructs 



6 



dimensionality at the item level because an item may measure a property that is lost in the 
analysis of parcels (Dorans & Lawrence, 1987). Therefore, the parcel approach may not be 
appropriate if one is interested in exploring the item-level structure of the test through factor 
analysis. 

A second alternative for factor analyzing dichotomous variables was developed by Bock 
and Aitken (1981) and is generally referred to as full-information factor analysis. It is so named 
because the approach makes uses of the dichotomous responses to the items as well as their item 
response theory (IRT) parameters. Essentially, this procedure is based on Thurstone’s (1947) 
multiple-factor model where an unobservable response pattern, Yy, for person i and item j, is a 
linear combination of m normally distributed latent variables, G, weighted by the factor loadings, 
a 

( 1 ) Yy = a jX G u + a j2 0 2i +,. ..,+a jm 6 m , + 8 i ; 

the parameters in the model (1) are estimated by marginal maximum likelihood and the EM 
algorithm (Muraki & Englehard, 1985). A routine for performing full-information factor 
analysis is available in TESTFACT 2 (Wilson, Wood & Gibbons, 1991) however, the current 
software places an upper bound limit on the number of factors that can be extracted and is not 
equipped to analyze both dichotomous and polytomous item responses simultaneously. 

McDonald (1967) proposed non-linear factor analysis, another alternative technique for 
factor analyses of dichotomous data that also expands on Thurstone’s (1947) multiple factor 
analysis model. McDonald’s factor analytic model assumes that there is a non-linear relationship 
between the items and the factors. Including polynomials in a modified form of Thurstone’s 
(1947) multiple factor analysis model captures the non-linear relationship between the items and 
factors. Non-linear factor analysis can be expressed mathematically by the following form 
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(2) y,=a*,+££<V>/’ (f=lA...,n) 

/= 1 P-1 

where y, is an examinee’s score on item i, 

f is the number of factors necessary to account for examinee test performance, 
s is the degree of the polynomial used to fit the model with each factor, and, 
au p is the factor loading of the z'th item on the /th factor for the pih degree element in the 
polynomial (Hambleton & Rovinelli, 1986). Essentially, non-linear factor analysis involves 
fitting a series of increasingly complex models, where models differ in complexity by the order 
of the polynomial used. Factor loadings are estimated from the model with the polynomial that 
best fits the data. Non-linear factor analysis is a well reasoned approach to assessing the 
dimensionality of a set of dichotomous test items however, the current software NOHARM 
(Normal Ogive Harmonic Analysis Robust Method) program (Fraser, 1988) is limited in 
situations where data are comprised of both dichotomous and polytomous responses. 

Still yet another alternative for factor analyzing dichotomous data involves using 
weighted least squares in LISREL 8.3 (Joreskog & Sorbom, 1999), a procedure that is based on 
the asymptotically distribution free method developed by Browne (1984). The first step to 
implementing this procedure involves using PRELIS 2.3 (Joreskog & Sorbom, 1999) to compute 
estimates of the asymptotic variance and covariances of estimated tetrachoric, polychoric or 
polyserial correlations among model indicators (Joreskog & Sorbom, 1996). The inverted 
asymptotes are used as a weight matrix, W 1 , during minimization of the fit function 

(3) F{9) = (s-crjw- x (s-(j) 

where s' = (su, S21, S22, S31, ..., Skk) (Joreskog & Sorbom, 1996). LISREL minimizes the fit 
function (3) to estimate model parameters and fit indices. Preliminary research (Hu, Bentler, & 
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Kano, 1992) shows that the asymptotically distribution free method offers highly accurate 
estimates of x 2 , and other % 2 dependent tests (e.g. RMSEA, NCP, etc.) under conditions where 
sample sizes are large (> 5,000). 

For several reasons, weighted least squares was the confirmatory factor analytic method 
implemented in the current study. First, practical and substantive issues limited the feasibility of 
employing the parcel approach. Since, on average, eight items were associated with the factors 
in the current model, parcels with “approximately equal means and variances” as well as equal 
representations of “easy, middle, and hard items” could not be reasonably constructed (Dorans & 
Lawrence, 1999). Moreover, due to the fact that the current study is concerned with the item- 
level structure of the exam, and not the macro, parcel-level structure, this approach was not 
appropriate. Second, computer programs for full-information factor analysis and non-linear 
factor analysis, TESTFACT 2 (Wilson, Wood, & Gibbons, 1991) and NOHARM (Fraser, 1988) 
respectively, are not equipped to deal with dichotomous and polytomous items simultaneously. 

In addition, the upper bound limit on the number of factors that can be extracted using 
TESTFACT 2 prevents testing the current model. Third, under conditions where there is a large 
sample, the weighted least squares procedure offers a sufficient solution to the methodological 
problems associated with factor analyzing tests comprised of dichotomous and polytomous 
items. 

It was not possible to fit a CFA model that took into account the 100 exam indicators and 
13 latent factors using weighted least squares. An extreme computational burden is associated 
with the weighted least squares procedure as the number of items increases because the method 
requires generating and inverting the asymptotic variance/ covariance matrix of the estimated 
tetrachoric correlations (Bock, Gibbons & Muraki, 1988). Because the asymptotic variance/ 
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covariance matrix is of the order u * u (where u is the number of unique elements in the sample 
correlation or variance/ covariance matrix), and has u(u + l)/2 distinct elements (Joreskog & 
Sorbom, 1996), its size increases to an unwieldy degree as the number of items increases. For 
example, a model with 100 indicators would have a weight matrix of 12,753,775 distinct 
elements! As a result 13 separate models, one for each latent factor were fit to the data. 

Method 

Participants 

A total of 28,013 AP psychology examinees comprised the current data set. The 
examinees were 9,364 (33.4%) male and 18,648 (66.6%) female students enrolled in AP 
psychology classes at secondary schools in the continental United States of America or an 
associated territory. All of the examinees received a score on the exam. Of the examinees, 
70.6% were 12th graders, 27.2% were 1 1th graders and 2.1% were 10th graders. The ethnic 
composition of the participants was 0.5% American Indian or Alaskan Native, 1 .8% African 
American or Black, 4.7% Asian, Asian American or Pacific Islander, 17.1% Hispanic or Latino, 
67.4% White and 3.5% other; 6.1% of the examinees did not provide an ethnic description. 
Materials 

The AP exam in psychology contains a 70-minute section with 100 multiple choice 
questions and a 50-minute free-response section containing two essay questions. Item analysis 
revealed that two items were not functioning properly and were dropped from the test. The 
multiple choice and free response sections are designed to complement each other and to meet 
the overall course objectives and exam specifications (The College Board, 1999b). Multiple 
choice and free-response items are considered to be highly reliable. 
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Procedure 

A series of one factor CFA models were fit to the data using LISREL 8.3 (Joreskog & 
Sorbom, 1999). As mentioned, since both dichotomous and polytomous responses were 
observed, fitting these models to either the phi correlations or covariances would have biased the 
results obtained. Therefore, the models were fit to tetrachoric and polychoric correlations among 
items as well as their asymptotic variances and covariances using a weighted least squares 
procedure (Joreskog & Sorbom, 1996). These correlations and asymptotes were estimated using 
PRELIS 2.3 (Joreskog & Sorbom, 1999). 

Based on several recommendations (Hu & Bentler, 1999; MacCallum & Austin, 2000) 
and standards within the field, a model was considered to provide an acceptable fit to the data if 
the root mean square error of approximation (RMSEA) was less than (0.05), the right endpoint of 
the 90% confidence interval for RMSEA was lower than (0.08), the standardized root mean 
. square residual (SRMSR) was low (e.g. less than 0.10) and non-normed fit index (NNFI) was 
sufficiently close to (1.0) (e.g. > 0.90). It was not appropriate to use chi square x 2 values as an 
index of model fit since analyses were performed on the entire population of test takers and not a 
random sample. 

Because the current confirmatory factor analysis involved a strictly confirmatory 
situation in which a single formulated model is either accepted or rejected, it was not necessary 
to evaluate the plausibility of other competing models (Joreskog & Sorbom, 1993). 

Results and Discussion 

Values on fit indices for every model demonstrated acceptable fit unambiguously (see 
Table 1). Therefore, CFA models can be considered tenable representations of the underlying 
structure of the unidimensional factors of which the test is comprised. Model parameter 
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estimates of factor loadings were high in each model (see Figures 1-13). Most factor loadings 
were above (0.4) and only two out of the 1 10 path coefficients estimated were less than (0.2). 
Error variances associated with model indicators tended to be high (see Figures 1-13). In fact, 
practically every model fit to the data tended to yield estimates of error variances for each item 
that were higher than factor loadings associated with those items. This tendency does not 
necessarily imply poor model fit. It is more likely that the larger error variances reflect the 
unreliability of indicators comprised of a single item. Nevertheless, this result is perplexing and 
requires further attention. 

The current results are limited in several ways. First, since it was not possible to fit the 
entire 13-factor model to all of the exam indicators, we cannot comment on whether the entire 13 
factor model is supported empirically. Instead, we may only conclude from fit indices that the 
items associated with each factor appear to be unidimensional. While the finding is important in 
itself, it does not elucidate the nature of the entire factor structure of the exam. Despite this 
limitation, we can deduce, from the high factor loadings associated with each model fit to the 
data, that the items are highly related to latent variables in the models, which supports the 
construct validity of the exam. 

Several recommendations for future research are warranted in light of the results of the 
current investigation. First, a factor model that accounts for the entire test structure should be fit 
to the data. This first step requires either a new algorithm for the weighted least squares 
procedure that can perform well under situations where the number of indicators is large, or a 
new method altogether. These concerns were voiced by Muthen (1984) but have yet to be 
addressed adequately. 
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Second, while the current results support the construct validity of the 1 3 factors measured 
by the 1 999 AP Psychology Examination, it remains unseen whether the test contains items that 
encompass all of the factors relevant to the study of introductory psychology. In that respect, it 
would be interesting to undertake an extensive review of material related to introductory 
psychology in order to explore whether there are any relevant content domains missing from the 
test. This review might be accomplished by surveying introductory psychology professors and 
AP psychology teachers about the about the content domains they feel are most relevant to 
introductory psychology or by exploring the topics covered in introductory psychology 
textbooks. 

It would also be interesting to explore the cognitive dimensions measured by the items of 
the AP Psychology exam and provide diagnostic feedback to students on cognitive skills. 
Advanced quantitative methods like multidimensional item response theory models MIRT (see 
Embretson & Reise, 2000), regression trees (Sheehan, 1997) and the rule-space model (Tatsuoka 
& Tatsuoka, 1992) are available for addressing research questions of this kind. Implementation 
of these methods offers not only greater insight into the psychometric properties of the test and 
its construct validity, but can also be a means for providing information about cognitive skills 
and instructional feedback to examinees- a worthy end. 
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Table 1 



Goodness of Fit Statistics for Each 1 -Factor Model Fit to the Data 



Model 


Model Label 


No. 

items 


RMSEA 


90% 

Confidence 
Interval for 
RMSEA 


Standardized 

RMR 


NNFI 


1 . 


Methods, history, & 
approaches 


8 


0.017 


0.015; 0.019 


0.023 


0.99 


2. 


Biological bases of behavior 


11 


0.027 


0.025; 0.028 


0.034 


0.99 


3. 


Sensation & perception 


9 


0.024 


0.022; 0.026 


0.031 


0.98 


4. 


States of consciousness 


4 


0.007 


0.000; 0.015 


0.006 


1.00 


5. 


Learning 


8 


0.019 


0.017; 0.021 


0.018 


1.00 


6. 


Cognition 


11 


0.028 


0.027; 0.030 


0.038 


0.98 


7. 


Motivation & emotion 


9 


0.013 


0.011; 0.015 


0.019 


1.00 


8. 


Developmental psychology 


9 


0.030 


0.028; 0.032 


0.037 


0.98 


9. 


Personality 


8 


0.020 


0.017; 0.022 


0.022 


0.99 


10. 


Testing & individual 
differences 


6 


0.015 


0.012; 0.018 


0.016 


1.00 


11. 


Abnormal psychology 


8 


0.017 


0.014; 0.019 


0.019 


0.99 


12. 


Treatment of psychological 
disorders 


6 


0.032 


0.028; 0.035 


0.031 


0.99 


13. 


Social psychology 


9 


0.021 


0.019; 0.023 


0.024 


0.99 



Note. RMSEA = root mean square error of approximation; Standardized RMR = Standardized 



root mean square residual; NNFI = non-normed fit index. 
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Figure Captions 

Figure 1. Factor loadings and error variances for model 1 : Methods, approaches, and history. 
Figure 2. Factor loadings and error variances for model 2: Biological bases of behavior. 

Figure 3. Factor loadings and error variances for model 3: Sensation and perception. 

Figure 4. Factor loadings and error variances for model 4: States of consciousness. 

Figure 5. Factor loadings and error variances for model 5: Learning. 

Figure 6. Factor loadings and error variances for model 6: Cognition. 

Figure 7. Factor loadings and error variances for model 7: Motivation and emotion. 

Figure 8. Factor loadings and error variances for model 8: Developmental psychology. 

Figure 9. Factor loadings and error variances for model 9: Personality. 

Figure 10, Factor loadings and error variances for model 10: Testing and individual differences. 
Figure 1 1 . Factor loadings and error variances for model 1 1 : Abnormal psychology. 

Figure 12. Factor loadings and error variances for model 12: Treatment of psycholgical 
disorders. 

Figure 13. Factor loadings and error variances for model 13: Social psychology. 




19 



Introductory Psychological Constructs 



19 





20 



Introductory Psychological Constructs 



20 





21 



Introductory Psychological Constructs 



21 





22 



Introductory Psychological Constructs 



22 





23 



Introductory Psychological Constructs 



23 





24 



Introductory Psychological Constructs 



24 




O 

ERIC 



25 



Introductory Psychological Constructs 



25 





26 



Introductory Psychological Constructs 



26 





27 



Introductory Psychological Constructs 



27 





28 



Introductory Psychological Constructs 



28 





29 



Introductory Psychological Constructs 



29 





30 



Introductory Psychological Constructs 



30 




0 



31 



Introductory Psychological Constructs 3 1 





32 




U.S. Department of Education 

Office of Educational Research and Improvement (OERI) 
National Library of Education (NLE) 
Educational Resources Information Center (ERIC) 



EHICI 

TM033497 



REPRODUCTION RELEASE 

(Specific Document) 



I. DOCUMENT IDENTIFICATION: 



Title: W6 

[yJ HAT A7 VE 


L&DG6 Of 
7>ftr /vervSforr 


j»vT>w pu orv 

OObJ srvt^j crS 7 


PSVC4H>t'06\ < \ 




Author(s): 0 I^GriP 


<3 • Mruf WSKI' & 


THA AOS PAvsirS 






Corporate Source: (f*****^ 




&o**4> 


^ Publication Date: 

/\ VQVS-T , 7.00! 





II. REPRODUCTION RELEASE: 

In order to disseminate as widely as possible timely and significant materials of interest to the educational community, documents announced in the monthly 
abstract journal of the ERIC system, Resources in Education (RIE), are usually made available to users in microfiche, reproduced paper copy, and electronic 
media, and sold through the ERIC Document Reproduction Service (EDRS). Credit is given to the source of each document, and, if reproduction release is 
granted, one of the following notices is affixed to each document. 

If permission is granted to reproduce and disseminate the identified documents, please CHECK ONE of the following three options and sign at the bottom 
of the page. 

The sample sticker shown below will be The sample slicker shown below will be 

affixed to all Level 1 documents affixed to all Level 2A documents 

PERMISSION TO REPRODUCE AND 
DISSEMINATE THIS MATERIAL IN 
MICROFICHE, AND IN ELECTRONIC MEDIA 
FOR ERIC COLLECTION SUBSCRIBERS ONLY, 

HAS BEEN GRANTED BY 



TO THE EDUCATIONAL RESOURCES 
INFORMATION CENTER (ERIC) 

2A 



Level 2A 

□ 

Check here for Level 1 release, permitting reproduction Check here for Level 2A release, permitting reproduction Check here for Level 2B release, permitting reproduction 

and dissemination in microfiche or other ERIC archival and dissemination in microfiche and in electronic media for and dissemination in microfiche only 

media (e.g.. electronic) and paper copy. ERIC archival collection subscribers only 

Documents will be processed as indicated provided reproduction quality permits. 

If perrrission to reproduce is granted, but no box is checked, documents will be processed at Level t . 



PERMISSION TO REPRODUCE AND 
DISSEMINATE THIS MATERIAL HAS 
BEEN GRANTED BY 



TO THE EDUCATIONAL RESOURCES 
INFORMATION CENTER (ERIC) 



Level 1 



The sample sticker shown below will be 
affixed to all Level 2B documents 



PERMISSION TO REPRODUCE AND 
DISSEMINATE THIS MATERIAL IN 
MICROFICHE ONLY HAS BEEN GRANTED BY 



TO THE EDUCATIONAL RESOURCES 
INFORMATION CENTER (ERIC) 

2B 



Level 2B 

□ 



Sign 

here, 

please 



3 

ERIC 



I hereby grant to the Educational Resources Information Center (ERIC) nonexclusive permission to reproduce and disseminate these documents 
as indicated above. Reproduction from the ERIC microfiche or electronic media by persons other than ERIC employees and its system contractors 
requires permission from the copyright holder. Exception is made for non-profit reproduction by libraries and other service agencies to satisfy 
information needs of educators in response to discrete inquiries. 



Signature: 



‘A. 



Organizali on/Address: THe 

& PeiCUiPMertf, 

A/ V 1 602 3 



CO\-UeG€ /L€S&rtlMi 

HT CovuMQus AS* /Veu) 'iO<uC l 



& lcttvaJ aa-sveujSlcT / A&xS VM 



Printed Name/Position/Title: 



Telephone- 






E-Mail Address: /O 



FAX: 



(iiiV f Y2 7 



Date: 






6 ! 









III. DOCUMENT AVAILABILITY INFORMATION (FROM NON-ERIC SOURCE): 

If permission to reproduce is not granted to ERIC, or, if you wish ERIC to cite the availability of these documents from another source, 
please provide the following information regarding the availability of these documents. (ERIC will not announce a document unless it 
is publicly available, and a dependable source can be specified. Contributors should also be aware that ERIC selection criteria are 
significantly more stringent for documents that cannot be made available through EDRS.) 



Publisher/Distributor: 


Address: , 




*.«•*; ' 1 
1 ’ V 

i s 


• '.it*' V».’ »* 


t* .A A a A ' *„ ;1 ,1 .* 




Price: 


\ Vv* 


, «.<- ’4 V » \ 

n n * j .»■ .* 


u*/ 


\ iV. # tj r ( . >v 


y ■> 

> * e .*•; * i 



IV. REFERRAL OF ERIC TO COPYRIGHT/REPRODUCTION RIGHTS HOLDER: 



If the right to grant this reproduction release is held by someone other than the addressee, please provide the appropriate name and 
address: 



Name: 


Address: 






V. WHERE TO SEND THIS FORM: 


Send this form to the following ERIC Clearinghouse: ERIC Counseling & Student Services 

University of North Carolina at Greensboro 
201 Ferguson Building 
PO Box 26171 

Greensboro, NC 27402-6171 


* 






